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Abstract 

lighf-field  pravidr  a  vimal  stnst  ^  prestnet  by 

proditcins  a  firil-parallax  ^JT^^€-elEmmJPi^aI  ^£rial  rirfHal 

pertray'ied  IIt6}ec^  rnffter  rAu'l  dgpfA 

snd  ihat  carr  be  en^^ed  na^raUy  and  m^idlivrfy'.  Thb 
paper  dacumeEis  -e  comprehensire  hshi-Jfeld  di^lay  miem, 
TEcludins  campuEo^n,  phoEffnics,  and  ntferacdoE  sy^m 
eemp^nenSs,  ikai  fsJIjEKibie  and  scalable^  esinbhihm^  a  hnsis  for 
appbcalioE  Ee  fioEi]  inr^e-scale  eeHabarainre  tmd pormbSe^  mobile 
pro  dii  arc  frilertH  res. 

Author  Keywords 

Light-field;  holographic  displays;  3D  display;  bologr^liijc  videos 
hog-el^  holography,  iniegral  photography,  pleaDpric,  conaputed 
photography 

1.  Introduction 

Id  this  paper  we  pioseoi  the  results,  of  a  multi-phaEe  li^-field 
display  de^'elopnieat  eStm,  culminatiag  ia  the  ZScapeSl*  Modocr 
Display  C^'ZMD™'')  famih'  of  technologieE  and  prototypes. 
Achie^^  goals  of  rhe  eSon  iadude: 

■  Modnlarin^  ca  Nippon  multiple  di^lay  scales,  oiieatarions 
arid  configuratioDS,  up  to  6- foot  (1  .£  meter)  diagonal 

■■  FuU-paialMx,  omni-visibility^  for  correct,  cialiiial.  da-glasses 
3D  fiom.  all  TiiFivmg  positioiLS 

*  Wide  viewidg-angle  ta  accommodate  many  simnltadeous 
users  and  OBtural  collal>oiation 

*  Rapid  update-rates,  to  real-rime  interactive 

»  Phy'skaJly-accesEfljle  imagery  to  enable  direct  natnral, 

gestniaL  touch,  andpeiipheral-based  mteracriod 
»  Con^jatLbiliry  with  a  wide  i^miety'  of  esdsting  3D  sofirwiare 
applications 

■-  Holographic  (hologram-like}  fideliry  and 

percaptibility 

■  Ease  of  maintenance,  self-contaided  compntatLon 

We  ontline  the  system  design  that  was  de^'eloped  id  this  effort,  the 
photonic  approaches  enplcHed  and  tliEir  rekv'ant  constraints, 
coQipulational  requirements  and  architectures,  and  appUcarions 
demonstrated  thus  far  with  the  prototype  systems. 

Lighf-fpeJd  Disp^a-ys:  Light-field  diqjlays  -comprise  a  dass  of 
ItuK-dimeoELOcial  visual  information  presentation  devices  capable 
of  -cr-eating  reaUstic  votumettic  images  of  subfect  matter  widi  all 
depth-cues,  inchiding  frill  (om n i- direc rional)  paiallav,  occlusion, 
eccocnmodaiion,  and  others,  o\’er  a  br-oad  cange  of  'Viewing 
posLtLans.  Such  displays  are  generally  not  Ihon^t  of  as 
immersh'e  (such  as  virtual  reality'  or  odier  binocular  systems 
reqmjing  file  ™wEr  to  wear  glasses,  gog^s  or  other  visual 
petipher alsX  bnt  rath^  as  eiocentrically- viewed  de^'keE,  offering 
the  ability  to  naturally'  collaborate  and  interact  with  the  images 
and  -with  o-ther  simnltaneous  viewers. 

Methods  fo  fvodtjfos  fighi-fiefds:  Various  approaches  can  be 
considered  when  designing  a  ti^t-fiieJd  di^la^'.  Diffiacrive 
holographic  based  ^coaches,  mvobing  rapid  recorfiing. 


reconsimetion,  and  erasure  cycling,  and  others  im’ohing 
computation  of  mreiferencepalreins  with  subsequent  reccns-truction 
using  hL^-resolnliDd  spadal  li^t  modulators  have  been  proposed 
and  prototyped'"^  Other  approaches,  based  on  integral  photography' 
C'TP'^  involving  conYerting  hd^-densiy  pried  inJbamadon  to 
anffulnrfy-variant  arrays  of  collimated  ray  bundles  also  have 
been  demoostrated.^  Both  of  these  apptoachjes  make  use  of  the 
conc^  of  a  ‘hoger  or  'fiologr^Mc  elemenf  as  the  bask  optical 
dement  of  the  light-fiidd  display.  Here  we  detail  the  ZScape^ 
Morion  Di^lay  [ZMD™)  lighr-field  di^lay'  sysiem  that  coiq)le5  a 
hybrid  of  refinctive  and  Ught-modulacing  eJemenls  'with  a  novel 
system  design  and  compnlatiocial  approaches  to  produce  a 
con'^inang  -dynamic  v-blumetnc  image  that  enables  nalnraJ 
perc^tion,  interaction  andcollaboiarion. 


Figure  1:  System  diagram  fnf  a  mcHdular  light-'ftEki  system. 
Software  and  peripheral  modules  in  blue. 


2.  System  Design 

Multiple  system  modules  were  considered  in  the  design  of 
ZMD™  (see  Figure  1).  The  Iriremcaon  and  Application 
subsystem  con^nses  the  area  ia  'a'hich  the  'user  views  and 
manipulates  the  light-field  image.  The  Photonics  Module, 
provides  the  images,  comprising  q^atial  li^t  modulator(sX!SLMs) 
and  hogel  aira^"  optics.  '5i'here  digilai  infijtmation  is  converted  to 
Ughr  in  the  system.  The  Con^ulahon  and  Distiiburi-Dn  layer  feeds 
the  Photonks  subsystem,  com^erling  abstract  3-D  representations 
of  data  to  piael  information,  and  pro\iding  that  digital  information 
to  (ofien  m'uLtiple}  spatial  Ught  modulatots.  The  Conpilation 
system  also  indudes  a  separate  off-the-shelf  (OTS)  'woihstition 
that  hosts  commercial  OTS  and  propiietaiy  3D  sofba'are 
applications.  A  propiietaiy  software  and  firmware-based 
opetacing  system,  called  “Argon,”  integrates  all  of  these  physical 
elements.  Argon  comprises  plug-in  modules  that  govetn  3D  data 
interception,  dam  -distribution  and  synchronization,  hogd 
information  rendering,  and  display  caMbiatLon. 

3.  ZMD™  Light-field  Photonics  Approaclies 
Phofonros  Des^n:  To  produce  votumetric  imagery  for  this 
wotis,  we  chose  to  use  a  modified  IP-based  approach  to  fight-field 
generation  that  converts  pivel  information  to  q'uasi-collimated, 
angularly'  distributed  light  beam  arrays.  In  this  approach,  a  fixed 
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eidslE  b^tn'&en  pixgl  deosiry,  licggl  Qptic  Edze.  aiLd 
hog-frl-cpdc  nunajerlcal  apemire,  determining  the  T.ienring  angle, 
i^a^ltctLGii,  and  nmninal  depdL  ’budgeft  of  the  display  {detailed  in 
Figure  2).  In  addition,,  because  SLMb  are  required  with  pusel 
sizes  less  rhan  SO  raicrcins  and  large  pl^skel  areas  (greater  rhan 
15  cm  diagaQaJ)j  inul%le  SLM  projectors  must  be  tiled.  Several 
tiling,  relay,  SLM  and  optkaJ  approecbes  were  consideTed  is  diis 
researck,  and  ti^o  partinilarly  mccessful  syatems  are  documented 
here. 


due  to  variable  retardaticin  across  LC  panels  and  relay  NA. 
Additionalty,  a  ligkt-source  and  komogenizer  are  also  integrated^ 
increasing  the  ptLyskal  length  af  tbe  system.  Color  is  achie^'ed 
through  separation  tenDporal  muMpleiing  at  120  Hz  w-idi  RGB 
LED  Ughr  sources,  lesaltmg  ia  a  color  hrame  rate  of  40  Hz  in  this 
design.  Lack  tile  unit  comprises  24  LC-based  modules, 
presenting  some  challenge  to  ackiebUng  colao-  and  brightness 
uniformity  and  comributiiig  to  some  arrifects  in  tke  resulting  hgkt- 
tield  image. 


Figure  2:  Key  basic  IP-hasod  light-field  hi^el  optic 
varia  btes  and  their  interrelaticHnships 

Approach  i:  OLED  +  Fiber  Tspers:  Dus  approach  integrates 
tiber  tapers  to  magnify  and  abut  the  output  fiom  multiple 
neighboring  organic  ligkt-emitting  diode  (OLED)  micro-displays, 
producing  a  bed  of  pizels  with  unifotm  emissian  numerical 
aperrure  (NA)  and  talecentric iry.^  Die  OLED  imoo-displays  are 
monockromatic  widi  resolution  of  ^0  by  dOO  piaels  and  a  relayed 
pusel  spacing  of  ^proiimately  31  microns,  determined  by  2X 
fiber  taper-based  magnification.  We  used  3mm-diameter  ^ss 
doublet  lenskts  as  a  refinactive  component  of  the  kogd  optic  array, 
witk  inter-lenset  absorbing  grid  to  mitigate  crosslaJE..  A  novel 
meckanical  stmcrure-was  designed  to  si^^port  the  OLED  displays, 
and  fiber  mpers  in  groins  of  sue,  and  allow  tor  mi-niiTnal  seam 
widths  between  neighboring  tiles  (see  Figure  3,  left). 


Airrcfcft  1  iirtrtMi  =  r&Ein'y 


p»b^ 

Figure  3:  Two  of  the  photonics  approach  es  developed  and 
demonatratod  in  present  FBSoar-ch. 

Approach  2.'  LC  +  Free-space  Opficai  Relsy:  Transmissive 
liquid  crystal  (LC)  panels  and  less-eEpensrve,  lifter-weight 
polymer  doublet  magnifying  relays  have  been  incotpotated  in 
another  approach,  dubbed  "XjENI”  (see  Figure  3.  center  and 
rift).  GENl  monoclirome  LC  parcels  witk  resolirtion  1920  by 
lOSO  and  piisel  size  of  £.5  microns  magnified  and  relay^ed  to 
pizels  approximately  19  microns  in  size.  A  reduciion  in  cost  and 
increase  in  lifetime  (due  to  n^lacement  of  rhe  OLED  panels  widi 
tong-life  LC)  compared  to  Approack  1  also  incurs  reduced 
telecentridty  and  some  vignetting  and  luminance  non-unitormity^ 


Fifing  Approach  to  ScaHng:  ZMD™  kas  been  designed  with 
a  tiled  approach  to  scaling,  to  enable  \mious  sizes  of  displays  to 
be  produced  wiih  common  componenis  and  a  simple  and  robust 
merhanicai  assembly  (see  Figure  4).  The  GENl  prototype 
di^lay,  based  on  photonics  Approach  2.  consists  of  9  tiles,  each 
providing  SOmegapiiels  and  34  by  72  1.6  mm  diameter  hogels.,  in 
a  54  cm  diagonal  assembly.  The  tile  (schematic ally  shown  in 
Figure  4  -  left)  rg^resents  tke  basic  field-replaceable  unit  {‘TRLT’) 
of  the  GENl  di^lay  system.  Arbinarily-sized  displays  can  be 
assembled  with  multiple  tiles,  each  of  which  attaches  to  a 
communic  adons  backplane  within  the  mechanical  suppon  chassis. 


Figure  4::  Self-contair»ed  tiles  assemble  mcKiularfy  to 
tn-eate  a  single  Zfi4D'™.  Multiple  B-tile,  54-om  diagonal 
prototypes  have-  been  MnstrutSed  ar>d  are  in  operation. 


4.  Computation  and  Software  Architecture 
Tke  GENl  ZMD™  system  architecrure  iategrates  a  sophisticated 
operating  system,  called  '^Argon,'^  for  host  application  data 
extraction,  internal  data  distribution  to  render  nodes,  lift-field 
ho gel-view  rendering,  data  distribution  and  synckronizadon  to 
SLM  buf6^,  display  calibration  and  interaction  deiiice  intertoce. 

Scene  DefiCffpdon  Jn  feroepdon  an  d  Rendering:  GraphLcai 
3D  infotmadon,  in  the  fbim  of  OpenGL  scene  descriptions,  is 
iniercepted  fi^m  the  computer  hosting  the  software  applicadon, 
and  transmined  ^ia  Eihemet  hnik  to  tke  ZMD™  deduce  in  real¬ 
time.  Within  ihe  ZMD™,  scene  information  is  buffered  to  all  of 
the  graphical  processing  units  (GFUs)  in  the  render  engine.  The 
GPUs  render  eadi  hogeJ  liew  witk  a  "^double- fimstum’'  model 
camera,  using  an  OpenGL-based  rendering  algorithm  The 
rendering  is  performed,  in  essence,  fi-om  the  modeled  display 
emission  sur&.ce  perspechv^,  and  thus  produces  kogel  views  in  a 
single  step  process,  plimi-naring  the  need  for  costly  and  memorv- 
iniensive  block -transform  post-processing  st^s  (see  Figure  5).^'^ 
Multiple  akematrves  for  hogel  rendering  kar^'e  been  tested,  with 
the  best  speed-quaUty -balanced  results  t-o-date  produced  with 
Ecan-lme  based  approaches.  In  the  9-tile  GENl  display  pictured 
in  Figure  4.  each  hogel  vtiew  image  consists  of  76  by  76  pixels, 
spaiming  BO  hy  90  degrees  of  lift-field  output  per  hogel  and 
each  GPU  in  the  system  is  tesponsible  for  genetating  201  tS 
indepen-dEnt  hogel  views. 
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Figure  5:  Principle  cf  -dDubfe-frustum  rendering  used  fbr  parallel  generation  of  light-field  data. 


Syfi^em  Compufafiona^  ^  rncfiiitfecf4/fe:  Argon  kas  l>Eeii 
■designed  for  mar^iim  fle^Tibility,  ac^oomnodathe  for  scading  the 
pkyEical  di^lay  and  the  mimbei  of  tendering  nodes.  aJtemative 
types  of  rendaing  algorirhrns,  multiple  forms  of  spadal  ti^t 
moduLadon,  aad  methods  of  display  calibratioiL  Figure  6  details 
Aigon  fimctioiis. 


Figiure  G;  Functkmal  achematic  of  the  ZldD  Argon 
computational  operating  system 


The  Aigon  scene-desciipdon  mteTteptor  capteies  the  3D  scene 
diaifn  by  the  host  ^plicadon  in  real-time  and  replicates  tke  scene 
informanoD  across  the  array  of  render  nodes  ^'Ttbin  the  di^lay 
o^vec  smndaid  Blkemet  In  this  mennej,  the  ZMD™  can  be 
updated  as  the  scene  is  modified  and  man^nlated  by  the  host 
^Ucadon  Real-time  intejacnoa  with  die  liglit-field  scene  can  be 
accoDDplished  by  registering  a  3D  I'O  device  to  the  ZMD  displej' 
space.  Inieracdons  with  the  -^Ttmal  3D  space  are  sdll  managed  by 
the  host  appUcadon  Thus,  any  LO  device  that  is  compadble  ^-idi 
the  host  software  application  and  the  host  computer  ma^'  be 
seamlessl>’  integrated  wifii  the  display  [see  Figure  7]. 


Figiure  7\  Multiple  internotion  peipherals  hafve  been 
integrated  with  ZMD,  indudmg  A:  trBckhalL  and  B:  a  Q 
degree-nf-freedom  optically- tracked  stylus. 

5.  Results  and  Measurements 

An  early  prototype  based  on  pkotonks  Approach  1,  is  pktnred  in 
Figure  Here,  150  monochrome  OLED  panels  are  tiled 

togedier  to  produce  a  35cm  diagonal  display  nith  a  90  degree 


field  of  -^riew.  The  horizontally-otienied  pTotot>pe  enables  omni¬ 
directional  viewing  from  360  degrees.  The  interacnve  image 
tpdate  rate  for  this  system  was  demonstrated  at  appro ximatel}'  5 
utilizing  2005-i'iniage  OTS  Apple  MacMini  (Intel)  CPUs  and 
gr^hics  processing  hardware.  Pre-generated  movie  sequences 
display  at  frame-rates  of  o^-er  1 5  Hz  on  the  system 


Figure  8::  (L}  Monochrome,  OLED-ba&ed  prototype  based 
on  Approach  1  photonics  and  off-shelf  ccunputation,  {R)  An 
aerial  image  is  demonstrated  by  moving  a  sheet  of  paper 
above  the  ZMD™  GENl  piofotype. 

The  9-tile  ZMD™  GENl  prototype,  pictured  in  Figures  4^  7  and 
S(R),  was  measured  for  performance  ov^r  a  nmnber  of  key 
specifrcations.  Resuks  of  diese  measurements  are  summarized  in 
Table  1.  The  GENl  design  fearures  modified  reference  design 
CPU.'GPU  conpirtation  mgines  and  field-programmable  gate 
array  fFKjA)- based  post-processing  and  hogel  data  disnibution. 


Table  1:  ZMD™  GENl  S-tile  prototype  specifications 


L^erai  Image  Size 

364™i  X  413(1111,  0.54m  diag. 

Viewing  Range 

SflO*;  uiewatie  from  4-sides 

Bh^itness 

~20GcdfrrF 

OorttiaSI 

TOcl 

Cofcr 

ful  coicr,  (fter  4,fiOQ  cciors 

UnfemHy 

Image  Depth 

±13&nTi 

Nominaj  resdution 

Z.Snm  pcirts  (average) 

Resohsbie  image  dements 

iBOd’orF 

CXitput  Range  (fnim  nonnai^ 

±45* 

Fonn  fader 

Hcrizontal  crientatien  {<^e) 

Image  Upd^  Rale 

>4  Hz  (pre-ccTTputed) 

Refresh  Rate 

t6  Hz  (sirrpie  conteni),  3l-b  average 

Interactive  ffe^ionse  Time 

<0.1  sec 

Active  Hogei  Yieid 

>99^ 

Weight 

taikg 

Power  consumption 

2.aw 
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Figure  10:  Light-field  imagery  and  applicatjoniE  produced  using  the  ZMD  9-Tile-  System.  A:  Medical  imagery  for  training 
applicadon,  B:  LIDAR  imagery  of  MIT  main  buildings,  C:  Mission  plannirtg  integrating  LIDAR,  photography,  and  CGI 
symbology,  D:  Real-time  telepresenoe  demonstration  with  15Hz.  scanning  LIDAR  system,  networik  transmission,  and  display 
on  the  9-Tile  System  at  5Hz  update  rates. 


6.  Discussion  and  Conclusion 

A  fleioble,  scalable,  inieiatm'e  li^t-fieM  diEplay  system  has 
been  demonstrated  ifitb  multiple  im^IemeiLtadoiis.  The  system 
has  l>BeD  intepaced  for  leseaidi  in  e  number  of  ^plicetioiL 
areas,  and  with  a  rrirmber  of  usage  paradipns.  Funire  woris  will 
focTLE  on  in^ir-o^'ement  of  optical  and  SLM  quality,  scaling  and 
optimization  for  other  nse-modalities  such,  as  mobile  and  sin^e- 
nser,  integration  of  altemath'e  approaches  for  ljght-5eld 
Tendering  and  hogel  data.  disrribntLon  architectmes,  and 
incorporntLon  of  emerging  altemad^'e  interaction  and  real-i±ne 
dara  c^ture  per^heraJ  systems. 

7.  Impact  of  This  Research 

The  deveJopmenr  has  required  in  a  successful  baseline  ljgbt-5eld 
system  design  and  multiple  redncdoos  to  practice  for  practical 
ligtit-fieM  di^ay  that  can  be  leadiJy  scaled,  directly-mtegrared, 
and  performance-op rimi  zed  for  particular  ^plications  and 
coEiti.guratLons.  The  system  presented  provides  a  complete 
capability  that  is  nnprecedented  in  the  literatiue.  Although  other 
researchers  ha^'e  presenied  papers  -on  what  appear  co  be  similar 
displays,  those  di^lays  presented  -do  not  appear  to  be  modular 
or  scalable,  nor  do  they  appear  to  ha^ie  scalable,  real-time 
operating  systems  that  enable  real-time  manqniLadon  of  a  large. 
ftjU-col-or  hght-field  or  direct  mtegration  with  arbitrary'  software 
^pUcatiociE  throng  use  of  a  scene  desci^don  interceptor 
module.  Finally,  unprecedented  real-time  h^t-tield  capture, 
nerworh  transfer,  and  display  has  been  demonstrated  in  the 
comezt  of  the  present  research,  offering  potential  for  11™ 
•■hologTaphic”  poitraj'al  of  real-world  scenes  and  subject  matter. 
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